A Proposal for Incremental Dialogue Evaluation

The SIS community has made progress recently toward evaluating SLS systems that deal with dialogue, but there is still considerable work that needs to be done in this area. Our goal is to develop incremental ways to evaluate dialogue processing, not just going from Class D1 (dialogue pairs) to Class D2 (dialogue triples), but measuring aspects of dialogue processing other than length. We present two suggestions; one for extending the common evaluation procedures for dialogues, and one for modifying the scoring metric. I N T R O D U C T I O N There is no single dialogue problem. By its nature, dialogue processing is composed of many different capabilities matched to many different aspects of the problem. It is reasonable to expect that dialogue evaluation methodologies should be multffaceted to reflect this richness of structure. Ideally, each new addit ion to the set of evaluat ion methodologies should test a different aspect of dialogue processing, and should be harder than the methodologies that came before iL We present two suggestions: one which extends the common evaluation procedure in order to test one new aspect of dialogues, and one which modifies the scoring metric. Difference: 1. Conversation is cooperative, but a game is competitive. 2. In chess, the goal is clear (checkmate), but in a conversational dialogue, the goal is less dear. 3. In a chess game, any state can be completely and concisely represented by a single board position; in a dialogue it is not known what comprises a state, nor how to represent it. Like the game tree for chess, the human/computer dialogue tree is enormous, as indicated in figure 1. There are usually hundreds or thousands of alternatives the human may produce. The number of responses the system can make is much smaller; some responses may be clearly wrong, but seldom is there a single "right" or "best" response (just as there is seldom a single such move in chess). Even when striving for the same goal, two different people are very likely to choose very different paths. An Analogy with Chess We as a community have been thinking about dialogue evaluation in terms of whether the systems we are building give the "right" answer (the one the wizard gave, or the one agreed upon by the Principles of Interpretation) at every step. We have been trying to come up with a methodology to measure whether our systems can reproduce the wizard's answers at each step of a lengthy dialogue. But is this a reasonable approach? Participating in a dialogue, whether between two humans or between a human and a machine, bears a striking resemblance to playing a complex game such as chess.